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ABSTRACT 

Most principals are aware that the No Child Left Behind Act 
of 2001 makes it mandatory for school leaders who depend on federal funding 
to select and implement programs that are based on scientific research. This 
publication reviews five documents that offer insights into what is meant by 
scientifically based research and help school leaders evaluate the scientific 
basis of practices and programs. In Senate testimony, Grover J. Whitehurst 
stresses the need for increasing the amount of scientific research in 
education. A report by the Comprehensive School Reform Program Office 
provides school leaders with guidance on the use of scientific research in 
initiating comprehensive reform efforts. In "Scientifically-Based Research," 
Stephen Raudenbush anticipates and addresses questions that educators might 
have about the nature and implications of scientific research in education. 
Evelyn Jacob and C. Stephen White edited a special theme issue of 
"Educational Researcher" on scientific research in education. (RT) 
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Scientifically Based Research 

Kon Beghetto 



ost principals are aware that 
the No Child Left Behind 
Act (NCLB) of 2001 
makes it mandatory for 
school leaders who depend on federal 
funding to select and implement pro- 
grams that are based on scientific 
research. 

The notion of scientifically based 
research in education is not new. Over 
the years, the federal government’s 
efforts to make education a more scien- 
tific field has steadily progressed firom 
the Cooperative Research Act of 1954 
to the creation of the National Institute 
of Education in the early 1970s, later 
absorbed by the Office of Educational 
Research and Improvement (OERI). 
Two recent federal programs^ the 
Comprehensive School Reform 
Demonstration Program and the 
Reading Excellence Act, emphasi 2 e 
research-based reforms. 




Ron Beghetto is an assistant professor 
of teacher education at the University of 
Oregon. He has conducted research on 
data-based decision-making, educational 
evaluation, and teacher development 



This article was commissioned by the 
ERIC Clearinghouse on Educational 
Management at the University of Oregon 



for publication by NAESR 



In the latest reorganization of the US. 
Department of Education, OERI has 
been replaced by the Institute of 
Education Sciences (EES). One of the 
institute’s first projects is the What 
Works Clearinghouse (available online at 
www.w-w-c.org/ about.html), designed 
as a resource for educational decision- 
makers in selecting programs and prac- 
tices based on scientific research. 

Although there are no simple rules for 
determining whether the research base 
of a particular program or set of prac- 
tices is scientific, useful guidelines and 
considerations have been developed that 
can assist principals and other school 
leaders in their decision-making. 

The five documents reviewed in this 
Research Roundup offer insights into what 
is meant by scientifically based research, 
provide important considerations for 
school leaders when evaluating the sci- 
entific basis of practices and programs, 
and highlight the complexities involved 
in determining what programs and prac- 
tices are most appropriate for a particu- 
lar school. 

Grover J, Whitehurst stresses the 
need for increasing the amount of sci- 
entific research in education. 

The Comprehensive School 
Reform Program 0///ce provides 
school leaders with guidance on the use 



of scientific research in initiating com- 
prehensive reform efforts. 

Stephen Raudenbush anticipates 
and addresses questions that educa- 
tors might have about the nature and 
implications of scientific research in 
education. 

Richard J, Shavelson and Lisa 
Towne provide a National Research 
Council primer on scientific research in 
education. 

Evelyn Jacob and C. Stephen 
White edit a special theme issue of 
Educational Researcher on scientific 
research in education. 

Whitehurst, Grover J. “Statement 
of Grover J. Whitehurst, 

Assistant Secretary for Research 
and Improvement, Before the 
Senate Committee on Heaith, 
Education, Labor and Pensions.” 
Washington, D.C.: U.S. Departnnent 
of Education, June 2002. Available 
online at 

www.ed.gov/offices/iES/speeches 

In Senate testimony, Grover 
Whitehurst, director of the U. S. 
Department of Education’s reorga- 
nized Institute for Education Sciences, 
focuses on the federal government’s 
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commitment to scientifically based 
research in education; explains why 
the current emphasis on this type of 
research is important, and identifies 
key areas of research within the 
Department of Education that are in 
need of change. 

Whitehurst explains that adherence 
to scientifically based research will be 
a critical factor in the funding deci- 
sions and endorsement of programs 
that fall under the purview of NCLB. 
He argues that the research base sur- 
rounding key educational issues is 
thin to nonexistent, and that learning 
how to read is the only major pro- 
gram area that has “a substantial and 
persuasive research base.” He claims 
that our lack of scientific knowledge 
in education “is masked by a 'folk 
wisdom’... [that] employs unsystemat- 
ic techniques... doesn’t demand scien- 
tific knowledge... is inefficient, and...is 
hit or miss.” 

Whitehurst draws analogies 
between the current state of educa- 
tion and the premodern “folk prac- 
tices” and “folk wisdom” of medicine 
and agriculture. He notes that signifi- 
cant advances in medicine and agri- 
culture have resulted from a reliance 
on science and clinical trials, and 
asserts “there is every reason to 
believe that, if we invest in the educa- 
tion sciences and develop mecha- 
nisms to encourage evidence-based 



About ERIC 

The Educational Resources Information 
Center (ERIC) is a national information 
system operated by the Institute of 
Education Sciences (lES). The ERIC 
Clearinghouse on Educational 
Management, one of 16 such units in 
the system, was established at the 
University of Oregon in 1966. 

Research Roundup is prepared by the 
Clearinghouse with funding from lES, 

U.S. Department of Education, under 
contract no. ED-99-CO-0011. No federal 
funds were used in the printing. 

[ERIC Clearinghouse on Educational 
Management, 5207 University of Oregon, 
Eugene, Oregon 97403-5207. Phone 
800-438-8841. 

Web site: http://erlc.uoregon.edu 



practices, we will see progress and 
transformation.. .of the same order 
and magnitude as we have seen in 
medicine and agriculture.” 



Comprehensive School Reform 
Program Office, Office of 
Elementary and Secondary 
Education, U.S. Department of 
Education. Scientifically Based 
Research and the 
Comprehensive School Reform 
(CSR) Program. Washington, 
D.C.: U.S. Department of 
Education, August 2002. 18 
pages. Available online at 
www.ed.gov/offices/OESE/ 
com p ref or m/ap pe ndc. pdf . 



Administrators who seek federal 
funds to help defray the cost of 
implementing comprehensive school 
reform are required to select a reform 
program that “has been found, 
through scientifically based research, 
to significantly improve the academic 
achievement of students. . .or has 
been found to have strong evidence 
that such a program will significantly 
improve the academic achievement of 
participating children.” 

This document provides school 
leaders with guidance on how to eval- 
uate the scientific basis of reform 
strategies, methods, and programs by 
applying the following criteria: 

Systematic and empirical 
Research should involve careful plan- 
ning, have a sound theoretical foun- 
dation, be grounded in observational 
and experimental data collected from 
multiple sources, and ensure that 
claims are supported by measurable 
evidence. 

Rigorous data analysis. 
Analyses of data should utilize appro- 
priate methods, address questions of 
interest, account for complexities of 
the data, and justify the general con- 
clusions drawn from the study. 

Reliable and valid data 
collection. Data should be collected 
professionally and consistently to 
ensure that repeated measurements 



under similar conditions produce sim- 
ilar results, and that the collected data 
measure the outcomes they were 
designed to measure. 

Strong research design. The 
research design should maximize the 
researcher’s ability to answer the ques- 
tions of the study and/or to test a 
hypothesis. 

Results that allow for 
replication. Studies should present 
results with sufficient detail to ensure 
that replication and extension studies 
can be undertaken, and that the 
results are accessible and useful to 
practitioners. 

Expert scrutiny. Research stud- 
ies should undergo quality control 
from independent evaluators, such as 
peer reviewers from a scientific jour- 
nal or an independent panel of 
experts. 



Raudenbush, Stephen. 
“Scientifically-Based Research.” 
U.S, Department of Education 
Seminar on Scientifically-Based 
Research, February 2002. 
Washington, D.C.: U.S. Department 
of Education. Available online at 
www.ed.gov/offices/OESE/esea/ 
research/ 



A key section of this paper (one of 
several presentations at a U. S. 
Department of Education seminar on 
scientifically based research that are 
available at the above Web site) 
addresses questions pertaining to sci- 
entific methods in educational 
research, including the following: 

Do studies have to use random assignment 
to be considered scientific? 

While random assignment is useful 
in establishing cause and effect 
because it controls for variables, a 
study can be considered scientific 
without it. However, the researcher 
must show that confounding variables 
were controlled and did not bias the 
outcome. 

Are randomised studies possible in 
education? 

Although randomized studies can- 
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not answer every educational ques- 
tion, such studies are useful and 
should be employed with greater fre- 
quency. Raudenbush highlights several 
current and ongoing studies as exam- 
ples, including the Tennessee class- 
si2t experiment, evaluations of school 
reform, and randomi 2 ed studies of 
vouchers. 

Does qualitative research play a role in 
making educational research more scientific? 

Qualitative research serves the 
important function of providing vivid 
descriptions of how and why pro- 
grams do and do not work. 

How can insights he combined from vari- 
ous kinds of inquiry? 

Drawing on medical research that has 
established the link between smoking 
and lung cancer, Raudenbush illus- 
trates how non-experiments (surveys), 
true experiments (animal studies), and 
qualitative studies (examining lung tis- 
sue) “created a new consensus among 
scientists who had previously dis- 
agreed that smoking causes lung 
cancer.” 

Is there a risk in unrestrained enthusiasm 
for scientific research? 

Raudenbush cautions against over- 
estimating what science can offer edu- 
cation, explaining that “scientific work 
can inform but never replace the 
judgment of policy makers, practi- 
tioners, and parents.” He favors rigor- 
ous peer review to help prevent the 
overselling of the promises of 
scienee. 



Shavelson, Richard J., and Lisa 
Towne (Eds.). Scientific Research 
in Education. Washington, D.C.; 
Committee on Scientific Principles 
in Education, National Research 
Council, 2002. 204 pages. 

Available online at www.nap.edu. 



This National Research Council 
report serves as a reasonably balanced 
and accessible primer on the topic of 
scientific research in education. The 
peer-reviewed report was generated 
by a diverse committee of prominent 
scholars who accepted the charge 
from the National Educational Policy 



and Priorities Board of the U. S. 
Department of Education to examine 
the nature and scope of scientifically 
based research in education and to 
consider how a federal agency can 
support and sustain such research. 

Several themes germane to educa- 
tional research are put forward in this 
report. By carefully considering these 
themes, educational leaders will be in 
a better position to understand the 
nature of scientific research in educa- 
tion and determine the scientific basis 
of their present school programs and 
practices. 

General scientific principles. 
The authors argue that these six gen- 
eral principles characterize all scientif- 
ic research, including scientifically 
based educational research. 

• Poses significant questions that 
can be investigated empirically; 

• Links research to relevant theory; 

• Uses methods that permit direct 
investigation; 

• Provides a coherent and explicit 
chain of reasoning; 

• Replicates and generalizes across 
studies; and 

• Encourages professional scrutiny 
and critique. 

Accumulation of knowledge. 
The accumulation of scientific 
knowledge takes time, develops from 
diverse methods, relies on multiple 
studies across varied contexts, and 
complies with the norms and eviden- 
tiary standards of a scientific com- 
munity. Conclusions regarding the 
causes and effects of any particular 
program or practice can rarely, if 
ever, be made on the basis of a single 
study. 

Defining a scientific study. 
The design of a study is not sufficient 
for considering whether or not it is 
scientific. Rather, the scientific nature 
of a study is determined by its align- 
ment with the general scientific princi- 
ples outhned above and its adherence 
to the rigorous, self-regulated norms 
of a scientific community. 

The nature of education. 
Because education is a complex, 
diverse, and value-laden field, it is 



important to carefully consider the 
unique features of a particular school 
context when determining whether 
theories and findings from research 
studies will meaningfully generaUze to 
meet specific needs. 



Jacob, Evelyn, and C. Stephen 
White (Eds.). Theme Issue on 
Scientific Research in Education. 
Educational Researcher 31, 8 
(November 2002), Available online 
at www.aera.net/pubs/er/toc/ 
er3108.htm 



A recent theme issue of Educational 
Researcher included reactions from sev- 
eral prominent educational 
researchers, representing a variety of 
approaches and perspectives, to an 
article written by three key contribu- 
tors to the National Research 
Council’s report on Scientific 
Research in Education. In that article, 
Michael Feuer, Lisa Towne, and 
Richard Shavelson assert that the 
recent federal requirements for scien- 
tifically based research create an 
opportunity to consider means of 
improving educational research. They 
presented four arguments in support 
of developing a “scientific culture of 
educational research” as a means for 
enhancing the scientific basis of edu- 
cational research: 

• Nurturing and reinforcing a scien- 
tific culture of educational research is 
a critical task for promoting better 
research. 

• Scientific culture is a set of norms 
and practices within an ethos of hon- 
esty, openness, and continuous 
reflection. 

• Individual researchers and 
research institutions have the respon- 
sibility for developing such a scientific 
culture. 

• A federal educational research 
agency and the American Educational 
Research Association (AERA) can 
and must play crucial leadership roles 
in fostering, supporting, and protect- 
ing a scientific culture among their 
grantees and members. 
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In their response, James Pellegrino 
and Susan Goldman see diversity 
within educational research as a 
strength and declare that “educadon- 
al research is often better served by 
the mulddisciplinary, researcher- 
pracddoner team approach.” They 
explain that one of the most impor- 
tant criticisms of educational 
research is that it often lacks a 
meaningful connection to practice. 
They assert that educational 
researchers must include practition- 
ers in their research community so 
as to better understand and address 
problems of practice. 

David Berliner focuses on the 



complexities inherent in conducting 
research in education. For example, 
he notes that teachers' characteristics 
affect and are simultaneously affect- 
ed by student and environmental 
characteristics. He also observes that 
changes in social and cultural envi- 
ronments can render findings from 
prior scientific research irrelevant. 

Frederick Erickson and Kris 
Gutierrez explain that focusing only 
on “what works” can blind educa- 
tors to important aspects and unan- 
ticipated outcomes of education 
processes. The authors admonish 
educational researchers and the con- 
sumers of research to develop a 



critical and realistic stance toward 
science. They stress the need for 
explicitly recognizing the complexity 
inherent in conducting educational 
research and the importance of mul- 
tiple methods and perspectives as a 
means for promoting “positive edu- 
cational change.” 

Elizabeth Adams St, Pierre criti- 
cizes the NRC report’s rejection of 
postmodernism and the diversity of 
approaches included within that cat- 
egory. She argues for the need “to 
keep educational research an open 
field of play in which science does 
not obstruct but enables the prolif- 
eration of knowledge.” □ 
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Classroom Imsfirudioo Ttiiat Works: 

Research°iased Strategies for Increasing Student Achievement 

by Robert J. Marzano, Debra J. Pickering, and Jane E. Pollock 

What works in education? How do we know? How can teachers find out? How can educational research find 
its way into the classroom? How can we apply it to help individual students? Questions like these arise in 
most schools and busy educators often don’t have time to find the answers. The authors have examined 
decades of research findings and distilled the results into nine broad teaching strategies that have positive 
effects on student learning. Each chapter presents extended classroom examples of teachers and students 
in action; models of successful instruction; and many frames, rubrics, organizers, and charts to help 
teachers plan and implement the strategies. 
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